[VL] Deserialize broadcast build side per the consuming stage's cuDF tag by ReemaAlzaid · Pull Request #12838 · apache/gluten

ReemaAlzaid · 2026-08-20T12:14:01Z

What changes are proposed in this pull request?

BuildSideRelation#deserialized built its Runtime from the session conf, so
broadcast batch residency was decided session wide, while the consuming
stage's value stream contract is decided per stage by the cuDF tag. The two
disagree in both directions: an untagged stage receives device-resident
batches on a host contract ValueStreamNode (TPC-H q16), and a tagged stage's
CudfHashJoinBuild requires device input (TPC-DS q95).

Pass the consuming stage's offloadCuda into deserialized instead, so a
cuDF offloaded stage gets the GPU serializer and a non-offloaded stage gets
the host one. This mirrors the shuffle read path, which already selects its
deserializer from the consumer's output type rather than the session conf.

BroadcastUtils keeps the no arg deserialized, which is now host-resident —
it feeds VeloxColumnarToRow and wants host batches in every mode.

How was this patch tested?

Tested on an L40S without #12471 applied: TPC-DS q95 and TPC-H q16 both pass
in hybrid and pure-GPU.

Was this patch authored or co-authored using generative AI tooling?

…uDF conf BuildSideRelation#deserialized creates its Runtime from the session conf, so with spark.gluten.sql.columnar.cudf=true VeloxRuntime::createColumnarBatchSerializer returns the GPU serializer and every broadcast batch is uploaded to the device, even when the consuming stage was not offloaded to cuDF and planned a host-contract value stream. TPC-H q16 reproduces this deterministically: its not-in subquery is a null-aware anti join, which Spark always executes as a broadcast join even with spark.sql.autoBroadcastJoinThreshold=-1, so the untagged consumer receives device-resident CudfVectors and fails. Pass a per-instance COLUMNAR_CUDF_ENABLED=false override so broadcast batches are always deserialized to host. Both serializer flavors share the same wire format (the GPU one only overrides deserialize), and cuDF consumers upload host batches themselves via CudfVectorStream.

marin-ma · 2026-08-24T15:06:25Z

The join stage of BHJ is supported on GPU. We shouldn't set spark.sql.autoBroadcastJoinThreshold=-1.

marin-ma

Based on my understanding, the gap lies in the VeloxGpuColumnarBatchSerializer is always called regardless of the hash join stage is executed on cpu or gpu. When cudf is enabled, the query fails when the hash join is executed on CPU, but can pass on GPU.

This change will cause the hash join to fail when it is executed on the GPU.

marin-ma · 2026-08-24T15:11:30Z

Tested tpcds q95 locally with a small dataset. Before this change it can pass, but with this change it fails.

GLUTEN_IT_JVM_ARGS=-Xmx5G sbin/gluten-it.sh queries \
      --local --preset=velox --benchmark-type=ds --error-on-memleak --off-heap-size=10g -s=1 --threads=1 --iterations=1 --decimal-as-double=true --enable-history --enable-ui --data-gen=once --extra-conf=spark.gluten.sql.columnar.cudf=true --extra-conf=spark.gluten.sql.debug=true --extra-conf=spark.gluten.sql.debug.cudf=true --extra-conf=spark.gluten.velox.buildHashTableOncePerExecutor.enabled=false --extra-conf=spark.sql.autoBroadcastJoinThreshold=100000 --queries=q95

ReemaAlzaid · 2026-08-25T21:56:36Z

Based on my understanding, the gap lies in the VeloxGpuColumnarBatchSerializer is always called regardless of the hash join stage is executed on cpu or gpu. When cudf is enabled, the query fails when the hash join is executed on CPU, but can pass on GPU.

This change will cause the hash join to fail when it is executed on the GPU.

I agree with you on this. CudfValueStream extends CudfOperator, so the adapter marks it as producing GPU output and inserts no CudfFromVelox after it grep -c from-velox on a q95 run is 0. The GPU serializer was the only thing uploading. This PR removes it and puts nothing in its place, so CudfHashJoinBuild gets a host batch and the cast fails. Exactly what you saw.

The missing piece is #12471, which uploads in CudfVectorStream instead. With both applied q95 passes, same plan and same build tables as before one H2D copy, just moved to the consumer

Also on autoBroadcastJoinThreshold=-1; I only used it to isolate q16 to a single broadcast, and I'll re run at the default.

So there are two options we could do:

Land [VL] Fix empty GPU broadcast joins and CudfVector host read crashes in the cuDF backend #12471 first and rebase this on it.
Or Make this self contained: pass the consuming stage's offloadCuda into BuildSideRelation#deserialized instead of a hard coded false. GPU stage → GPU serializer, CPU stage → host serializer, no cross PR dependency

I lean toward 2. Happy to do either

marin-ma · 2026-08-26T08:27:36Z

I wouldn't agree with the changes in #12471 Cudf related execution shouldn't be introduced into RowVectorStream. The errors should mostly come from the incorrect shape of the pipeline. We should fix the pipeline construction rather than adding the "RowVector from/to CudfVector" conversions in Gluten.

Please continue with solution 2. Based on the current support status, after the fix the correct pipeline for BHJ should be:

GPU stage:
GPU Deserialiser (host to device) -> CudfValueStream -> Cudf pipeline -> CudfToVelox (device to host) -> shuffle write/broadcast serialization/c2r/...

CPU stage:
CPU Deserialiser -> RowVectorStream -> CPU pipeline -> shuffle write/broadcast serialization/c2r/...

ReemaAlzaid · 2026-08-26T19:55:52Z

@marin-ma Pushed, the deserializer now follows the consuming stage's offloadCuda

Traced on q95: CudfValueStream → CudfHashJoinBuild on the build side, CudfValueStream → … → CudfToVelox on the probe, 0 CudfFromVelox inserted. Untagged stages take the host deserializer and RowVectorStream

One flag: dropping #12471 exposes q3/q10/q18, failing at CudfTopN::doAddInput TakeOrderedAndProject's runtime built exchange skips AdjustStageExecutionMode. A control run with the old serializer fails identically, so not a regression here, but it's the same wrong pipeline shape. Should I open a separate issue for it? or continue on it from the #12471 patch?

marin-ma · 2026-08-27T08:11:32Z

The change looks good. Can you help to add a unit test? Thanks.

marin-ma · 2026-08-27T08:19:45Z

One flag: dropping #12471 exposes q3/q10/q18, failing at CudfTopN::doAddInput TakeOrderedAndProject's runtime built exchange skips AdjustStageExecutionMode. A control run with the old serializer fails identically, so not a regression here, but it's the same wrong pipeline shape. Should I open a separate issue for it? or continue on it from the #12471 patch?

The idea of #12471 is slightly off track. If the failures in q3/q10/q18 are unrelated to BHJ, I’d suggest closing #12471 and opening a new issue/pr.

ReemaAlzaid · 2026-08-27T09:58:39Z

Thanks for the review @marin-ma
I have added the UT

marin-ma · 2026-08-27T10:40:55Z

+  }
+
+  test("GLUTEN-12838: broadcast build side follows the consuming stage's cuDF tag") {
+    // We need a broadcast that lands in a CPU stage. NOT IN gives us one for free: it


@ReemaAlzaid Thanks for adding the test. My understanding is that we need to test broadcast in a CPU stage + hash join in a CPU stage when cudf is enabled based on your issue description:

VeloxRuntime::createColumnarBatchSerializer returns VeloxGpuColumnarBatchSerializer and every broadcast batch is uploaded to the device even when the consuming stage was not offloaded to cuDF and planned a host-contract value stream.

Does this test cover this case?

Yes. I instrumented the test to check the tag directly:

bhj offloadCuda=false nullAware=true stage offloadCuda=false (both WholeStageTransformers in the plan)

So both the broadcast hash join and its stages are not offloaded, even with spark.gluten.sql.columnar.cudf=true. This is exactly the case where the session config was incorrectly selecting the GPU serializer and passing device-resident CudfVectors into a host side value stream.

I also confirmed the test catches the regression. Reverting the serializer selection back to the session config makes it fail with:

childAt: index < childrenSize_ (0 vs. 0)

on:

ROW<"node_value-stream:0_0":BIGINT, …>

because the CPU stage receives a device vector with no children.

With the fix, both tests pass.

Can you the tag check for BHJ to the test?

Added. The test now asserts the BHJ is not offloaded to cuDF

marin-ma

LGTM. Thanks for iterating on this!

github-actions Bot added the VELOX label Aug 20, 2026

ReemaAlzaid mentioned this pull request Aug 20, 2026

[VL] Fix empty GPU broadcast joins and CudfVector host read crashes in the cuDF backend #12471

Closed

marin-ma requested changes Aug 24, 2026

View reviewed changes

ReemaAlzaid and others added 2 commits August 26, 2026 12:18

Merge branch 'main' into cudf-broadcast-host-deserialize

f2245ec

[VL] Deserialize broadcast build side per the consuming stage's cuDF tag

b2a4d2b

ReemaAlzaid requested a review from marin-ma August 26, 2026 19:52

Merge branch 'main' into cudf-broadcast-host-deserialize

df3c23f

ReemaAlzaid changed the title ~~[VL] Deserialize broadcast build side on host regardless of session cuDF conf~~ [VL] Deserialize broadcast build side per the consuming stage's cuDF tag Aug 26, 2026

add unit test

5396118

marin-ma reviewed Aug 27, 2026

View reviewed changes

ReemaAlzaid requested a review from marin-ma August 27, 2026 13:22

[VL] Add regression test for broadcast build side residency

379c0ef

marin-ma approved these changes Aug 31, 2026

View reviewed changes

marin-ma merged commit f9c1152 into apache:main Aug 31, 2026
61 checks passed

Uh oh!

Conversation

ReemaAlzaid commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes are proposed in this pull request?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

marin-ma commented Aug 24, 2026

Uh oh!

marin-ma left a comment

Choose a reason for hiding this comment

Uh oh!

marin-ma commented Aug 24, 2026

Uh oh!

ReemaAlzaid commented Aug 25, 2026

Uh oh!

marin-ma commented Aug 26, 2026

Uh oh!

ReemaAlzaid commented Aug 26, 2026

Uh oh!

marin-ma commented Aug 27, 2026

Uh oh!

marin-ma commented Aug 27, 2026

Uh oh!

ReemaAlzaid commented Aug 27, 2026

Uh oh!

marin-ma Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ReemaAlzaid Aug 27, 2026

Choose a reason for hiding this comment

Uh oh!

marin-ma Aug 27, 2026

Choose a reason for hiding this comment

Uh oh!

ReemaAlzaid Aug 30, 2026

Choose a reason for hiding this comment

Uh oh!

marin-ma left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ReemaAlzaid commented Aug 20, 2026 •

edited

Loading

marin-ma Aug 27, 2026 •

edited

Loading